Co-operative Evaluation of a Desktop Virtual Reality System

A summative usability evaluation of a desktop virtual reality (VR) system was developed and a preliminary study then conducted. The purpose of the study was twofold. Firstly, to test whether the traditional evaluation technique, co-operative evaluation, is effective in the evaluation of desktop VR systems. Co-operative evaluation is a variation on a ‘think-aloud’ verbal protocol, whereby, in addition to concurrently 'thinking-aloud', users are encouraged to ask any questions about an evaluation, relating to the computer-based system, the application, or the tasks that they are required to perform during the evaluation. As well as this, the evaluator may ask questions of the user at any time during the evaluation. Results from the study indicate that this additional probing technique enables an evaluator to elicit further usability problems that may not have otherwise been exteriorized by the user. Additionally, a method is developed which attempts to turn round the qualitative 'think-aloud' type data into quantitative data. This provides a way of evaluating empirical 'think-aloud' evaluation methods and will be useful for comparing their effectiveness to evaluate 3D virtual reality systems. Secondly, attempts were made to capture a user’s awareness of the spatial layout of virtual environment (VE) and the objects and their positions within the VE. However, problems with the methods that were employed in the evaluation are identified, and suggestions that may inform future evaluations are made in an attempt to capture more directly a user's awareness of the spatial layout of a VE.